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ABSTRACT 



This empirical study compared a graduate online course with 
an equivalent course taught in a traditional face-to-face format. Comparisons 
included student ratings of instructor and course quality; assessment of 
course interaction, structure, and support; and learning outcomes such as 
course grades and student self-assessment of ability to perform various 
Instructional Systems Design (ISD) tasks. Results revealed that the students 
in the face-to-face course held slightly more positive perceptions about the 
instructor and overall course quality, although there was no difference 
between the two course formats in learning outcomes. The findings have direct 
implications for the development and delivery of online instruction, 
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Abstract: This empirical study compared a graduate online course with an equivalent course 
taught in a traditional face-to-face format. Comparisons included student ratings of instructor and 
course quality; assessment of course interaction, structure, and support; and learning outcomes 
such as course grades and student self-assessment of their ability to perform various Instructional 
Systems Design (ISD) tasks. Results revealed that the students in the face-to-face course held 
slightly more positive perceptions about the instructor and overall course quality although there 
was no difference between the two course formats in learning outcomes. The findings have direct 
implications for the development and delivery of online instruction. 



1. Introduction 

New advances in Internet-based technology have brought challenges and opportunities to education and 
training, particularly through online instruction. While online instruction is gaining popularity, it is not free from 
criticism. Many educators and trainers do not support online instruction because they do not believe it actually 
solves difficult teaching and learning problems (Conlon, 1997). Concerns include the changing nature of 
technology, the complexity of networked systems, unstable online learning environments, and the limited 
understanding of how much students and instructors need to know to successfully participate (Brandt, 1996). 
Online instruction also threatens to commercialize education, isolate students and faculty, reduce standards, and 
devalue university degrees (Gallick, 1998). While these concerns may be unwarranted, there is little research to 
fully understand the benefits and pitfalls of online instruction, especially when compared to face-to-face learning 
environments. Gaining knowledge about the processes and outcomes of online instruction as compared to face-to- 
face environments will help educators make more informed decisions about future online course development and 
implementation. 



1.1 Problem Statement 

Although the popularity of online programs has increased in recent years, the capabilities and efficacy of such 
programs have yet to be fully investigated. Most effort in this area has been devoted to program development and 
much of the research has anecdotal in nature. With little empirical knowledge of Internet-based education 
outcomes, the need for research in this area is not only timely, but also imperative. The primary purpose of this 
study was to compare an online course with an equivalent course taught in a traditional face-to-face format. 
Comparisons included student ratings of instructor and course quality; assessment of course interaction, structure, 
and support; and learning outcomes such as course projects and student self-assessment of their ability to perform 
various ISD tasks. 
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1.2 Research Questions 



This study was designed to answer the following research questions: 

1. What differences exist in satisfaction with the learning experience of students enrolled in online vs. face-to- 



face learning environments? 
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2. What differences exist in student perceptions of student/instructor interaction, course structure, and course 
support between students enrolled in online vs. face-to-face learning environments? 

3. What differences exist in the learning outcomes (i.e., perceived content knowledge and quality of course 
projects) of students enrolled in online vs. face-to-face learning environments? 



2. Background 

There are many assumptions about Internet-based education, most of which are positive and optimistic (Relan 
& Gillani, 1997). The many challenges associated with creating meaningful learning environments via the Internet 
(Hill, 1997) include how to meet the expectations and needs of instructors and students and how to design courses 
so they provide a satisfying and effective learning environment. From program developer and instructor 
perspectives, understanding these issues is critical for the development and implementation of quality online 
instruction. 

While few experimental studies have compared the effectiveness of online instruction to the more traditional 
face-to-face offering, two recent studies provide encouraging results for developers of online instruction. Schutte 
(1997) conducted a small scale experiment in which he divided a class of 33 students into a traditional section and 
a virtual section taught on the World Wide Web (WWW). Schutte’s results showed that instruction provided online 
can result in improved performance. LaRose, Gregg, and Eastin (1998) conducted a similar study that compared 
the learning outcomes of students in a traditional lecture section to course section that provided pre-recorded audio 
via the WWW along with detailed course outlines and related course pages accessed on the Web. Results showed 
that the Web group had test scores and student attitude ratings equal to those of the traditional section. While these 
types of quasi-experimental studies present methodological challenges (e.g., dealing with small sample sizes, the 
effect of prior knowledge, etc.), they do provide an important first step into better understanding the effect of online 
instruction on learning outcomes and student satisfaction. 



3, Method 

3.1 Subjects and Instructional Setting 

This study compared outcome data obtained from students enrolled in one of two versions of a graduate level 
instructional design course for human resource professionals. One version of the course was taught using a 
traditional face-to-face format while the other version of the same course was offered totally online, with no direct 
face-to-face contact between the instructor and the students. Both courses were taught by the same instructor using 
the same content, activities, and projects. Nineteen students, most of whom are pursuing a graduate degree in 
HRD, were enrolled in the on-campus course. Nineteen students were also enrolled in the online version of the 
course. These students are also pursuing a graduate degree in HRD through a degree program that is taught 
completely online. 



3.2 Instrumentation 

The Course Interaction, Structure, and Support (CISS) instrument was used to assess student’s perceptions of 
the quality of various instruction-related variables. This instrument is a hybrid of the Distance and Open Learning 
Scale (DOLES) and the Dimensions of Distance Education (DDE) (Harrison, Seem an, Behm, Saba, Molise, & 
Williams, 1991; Jegede, Fraser, & Curtin, 1995). CISS consists of 11 items for the dialog construct, 8 items for 
support, 8 items for course structure, and 4 items for transactional distance. 



3.3 Procedures 

All data were collected at or near the end of the semester. The on-campus students completed paper versions of 
the instruments while the online students completed equivalent online version of the instruments. All instrument 
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data were entered into a statistical analysis package for later analysis. Statistical analysis was conducted using 
independent sample Mests with a significance level of .05. 

The search for differences in learning outcomes between students enrolled in the online vs. face-to-face 
learning environments was conducted using two primary sources of data. The quality of a major course project was 
the first indicator of learning outcomes. Students in both courses were required to complete a training package that 
included all training materials and instructional aids as well as all student materials needed to conduct the 
training. A blind review process was used to evaluate the quality of the course projects. Three HRD doctoral 
students with instructional design experience were asked to independently evaluate each project in terms of the 
presentation quality, course organization, degree of detail provided, and overall quality. The reviewers were not 
told that the purpose of the review was to compare the two course formats and they did not know which projects 
came from online or face-to face sections. The reviewers rated each project on a four-point scale for each of the 
four quality characteristics. Analysis of variance was used to examine ratings of the projects using a significance 
level of .05. 

A self-assessment instrument was also administered at the end of the course. This instrument asked students to 
rate their level of comfort at performing various ISD tasks. A total of 29 items were developed from the course 
objectives. Individual Mests were conducted to examine differences between the groups on each of the task items. 



4. Results 

The following results are from comparisons of the face-to-face and the online students’ perceptions in the 
areas of satisfaction, course interaction, course structure, and support. Further analysis compares student learning 
outcomes in terms of course project quality and a self-assessment of their ability to perform various ISD tasks. 



4.1 Student Satisfaction 

Student satisfaction was assessed using two items on the CISS instrument that corresponded to the global 
items used by the university to evaluate all campus courses. These items asked students to rate, on a five point 
Likert scale (5 = exceptionally high rating), the overall quality of the instruction and the course. 

Both groups provided positive ratings, although the face-to-face group displayed more positive views than the 
online group. The instructor’s overall teaching effectiveness received a mean rating for the face-to-face group of 
4.21 (SD = .79) while the online course mean was 3.58 (SD = 1.07). A similar, though non-significant, difference 
was found for the overall course quality rating, with the face-to-face group (M = 4.32, SD = .73) providing a 
slightly more positive rating than the online group (M= 3.79, SD = .92), r(36) = 1 .94, p > .05. 



4.2 Perceptions of Course Interaction, Structure, and Support 

The CISS instrument assessed student perceptions regarding course interaction, structure, and support 
throughout the semester. Using a four point Likert scale, the students indicated the degree to which they Agree (4) 
or Disagree (1) with various statements. Overall, both groups of students had positive perceptions, with the face-to- 
face students having significantly more positive views for interaction and support. 



4,2,1 Student Interactions 

Interaction among the students was assessed using 5 items that represented characteristics of a learning 
environment that support student communications, shared learning experiences, teamwork, building a sense of 
community, and promoting an increase in student contacts. As shown in Table 1, there was a significant difference 
in the between the two course formats, ^(33) = 3.847,/? < .05. 

Students enrolled in the traditional face-to-face course had a more favorable opinion of the amount and type of 
interactions among the students. Analysis of the individual items revealed no difference in the amount of contact 
among the students but a significant difference in terms of communication with other students in the class, sharing 
learning experiences with other students, perceptions of a sense of community, and being able to work in teams. 
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4,2,2 Student Instructor Interactions 



There was a difference in the perception of interaction between the instructor and the students as assessed 
using items covering teaching style, interaction with the instructor, instructor feedback on student progress, and 
the instructor’s treatment of the students. Analysis of the individual items revealed a significant difference relating 
to students being informed about their progress in the course, student and instructor interactions during the course, 
and the treatment of the students in the course, with a lower rating by the online students. 



CISS Instrument Sections 


Face-to-Face* 


Online* 


t 


Student Interactions 


3.23 (.51) 


2,65 (,37) 


3.847** 


Student & Instructor Interactions 


3.11 (.49) 


2,74 (,41) 


2.455** 


Course Structure 


3.16 (.41) 


2,94 (.40) 


1,641 


Instructor Support 


3.17 (.43) 


2,75 (.53) 


2.690** 


Departmental Support 


2.15 (.56) 


2,66 (.46) 


-2,921** 



Note: *Group means determined using a 4 point Likert scale ranging from Strongly Agree (4) to Strongly Disagree (1). Mean 
value above 2,5 indicates a positive perception among the students, A positive t value indicates a more positive perception 
among the face-to-face students. Standard deviations in parentheses. 

< ,05 

Table 1: Perceptions of Course Interaction, Structure, and Support, 



4,2,3 Course Structure 

There was no difference in the variable that examined issues of students being allowed to work at their own 
pace, quality of the course syllabus, structure of class activities, organization of the content, student input in the 
topics selection, teaching methods, and student assessment. 



4,2,4 Instructor and Departmental Support 

Instructor support included comprehensiveness and usefulness of feedback, student encouragement, and the 
instructor being able to help students identify problem areas with their studies. The students in the face-to-face 
course rated the instructor significantly higher for instructor support than the online students. Analysis of 
individual items showed no difference in the amount of encouragement the instructor provided to the students. 
These differences relate to the characteristics of instructor feedback and the ability of the instructor to assist 
students to identify weaknesses in their course preparation. 

Departmental support included information the department provided to the students, inquiring about their 
learning needs, and providing a communication link between the students and the instructor. The online students 
rated the departmental support significantly higher than the students enrolled in the face-to-face course. Analysis 
of individual items showed no difference in the departmental staff inquiring about the student satisfaction with the 
services provided and the departmental staff serving as facilitators between the instructor and the students. The 
differences related to students being informed about the support services, and about their learning support needs. 



4.3 Student Learning Outcomes 

Although student perceptions are important, the ultimate indicator of course effectiveness is the degree to 
which students reach the objectives. The following analysis examines differences in the quality of the final course 
projects and a comparison of the students’ self-assessment of their ability to perform each element of the ISD 
process. 



4,3,1 Blind Review of Course Projects 
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A primary outcome of the instructional design course was the completion of a complete training package that 
served as evidence that students had gained the knowledge and skills required of instructional designers. Since 
some of the students choose to work together on this project, the number of projects produced does not match the 
class enrollments. A blind review process was used to evaluate the quality of the course projects to compare the 
outcomes across the two course formats. 

Overall, the thirty projects were rated very favorably {M = 3.43, SD = .60). The overall mean rating of the 
face-to-face class projects was 3.47 {SD = .60) and the mean rating for the online class projects was 3.40 {SD = 
.61). The difference in the project ratings for the two groups was not significant. 

4. J.2 Self-Assessment 

A self-assessment instrument assessed students’ reported levels of comfort at performing various instructional 
design tasks. Each task was rated on a four point scale from Very Comfortable (4) to Very Uncomfortable (1). 
Significant differences were found on only five of the 29 items on the self-assessment instrument. The online group 
felt more comfortable than the face-to-face group when distinguishing among various ISD models while the face- 
to-face group felt more comfortable performing a learner analysis, preparing a content analysis, writing 
instructional goal statements, and writing terminal objectives in comparison to the online group upon completion 
of the course. 

5. Discussion 

As discussed in the opening sections of this paper, the effectiveness of online instruction is unclear. The results 
of this study show that student satisfaction with their learning experience tends to be slightly more positive for 
students in a traditional course format although there is no difference in the quality of the learning. These results 
suggest that online instruction can be as effective as traditional face-to-face instruction. 

Students from both groups provided positive ratings of the instruction and the course. Although the face-to- 
face group provided a slightly more positive rating of instructor quality than the online group, the reasons for this 
difference are not evident. It is possible that the instructor was more effective in the traditional format, although 
the lack of difference in the learning outcomes does not support this. Another possible explanation is that student 
ratings may be higher when there is a personal connection between the instructor and the students, something that 
may not occur in an online course. Another possibility is that the response set of online students tends to be lower 
than the response set of students in a traditional format. Clearly, additional study of the influence of online 
instruction on student ratings is needed. 

Generally, the face-to-face students indicated a more positive perspective on the learning environment 
characteristics than the online students. Considering the fact that the face-to-face class met in person once a week 
for a 3 hour period throughout the semester, the differences in student interaction levels are to be expected. 
Students in face-to-face courses can more easily get together for an extended period of time to discuss class 
projects, work out any differences of opinion, and build social relationships. In contrast, online students do not 
have similar opportunities, although the technology provides a surrogate form for similar interactions. This 
suggests that the online environment may lack the strong social dimension that is beneficial to face-to-face 
classroom experiences. 

Differences between the online and face-to-face groups were significant for instructor and departmental 
support. Students in the face-to-face course reported higher levels of instructor support than did the online 
students. A more detailed item analysis reflected that the differences stemmed from the characteristics of instructor 
feedback. This makes sense in view of the differing contexts of the two classes. The face-to-face setting allowed the 
instructor to vary the nature and type of feedback as needed. In the online course however, the instructor feedback 
was limited largely to e-mail, fax, uploaded files, and periodic telephone conversations as a means of delivering 
feedback. The face-to-face students received live and dynamic forms of support from the instructor while the online 
group received support in a form of one way static communication. In terms of departmental support, the online 
students reported higher ratings than the face-to-face students. This difference is easily explained by the fact that 
the face-to-face class had direct contact with the instructor and a part time teaching assistant, therefore they had 
little need for support from the department. In contrast, given the complexities of online technologies, the online 
class had more need for technical support, a service that was provided by the department. 

The lack of difference in the learning outcomes from the two course formats supports the continued 
development of online programs. Using a blind review process to judge the quality of the major course projects, the 
ratings of three independent reviewers showed no difference in the quality of the projects across the two course 
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formats. While there were significant differences on five of the 29 self-assessment items, examination of the results 
as a whole indicate the students in both groups are equally comfortable in performing the instructional design 
tasks. Four of the sets of means fell between the “comfortable - very comfortable” range and one set of means fell 
between the “uncomfortable - comfortable” range. Overall, both groups indicated a level of comfort at performing 
the tasks. It is worth devoting some discussion as to why these few differences did exist. 

5.1 Implications for Future Online Programs 

The ultimate question surrounding online instruction is whether or not it is as effective as traditional face-to- 
face modes. The findings of this study show that online learning can be as effective as face-to-face learning in spite 
of the fact that students in online programs are less satisfied with their experience than students in more traditional 
learning environments. In view of these findings, several implications emerge pertaining to future online 
programs. 

First, this analysis suggests that the development and use of online programs should continue. Further 
examination of feedback and student progress are needed to improve overall student/instructor communication. 
This includes identifying and implementing new communication measures to facilitate student/instructor 
communication at appropriate points in the course. Second, a better understanding of why online learners report 
lower levels of comfort with their learning is needed so specific strategies for improving delivery of online 
programs that increase student confidence levels can be developed. Finally, educational practitioners who may 
enroll in or develop online courses need to be familiar with the limitations of online programs. Such an awareness 
will ensure that the expectations of learners are met and the intended course goals can be attained. For instance, 
the findings in this study suggest that online instruction may not be suitable for courses that require high degrees of 
student instructor interaction and feedback, such as performance-based training methods courses that rely on 
considerable mentoring and coaching. Until the technologies for online instruction better simulate real time 
interaction, program developers need to avoid courses that require frequent socialization between students and the 
instructor. 
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